Bounds for Fisher information and its production under flow 
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Abstract 

We prove that two well-known measures of information are interrelated in interesting and useful 
ways when applied to nonequilibrium circumstances. A nontrivial form of the lower bound for the 
Fisher information measure is derived in presence of a flux vector, which satisfies the continuity 
equation. We also establish a novel upper bound on the time derivative (production) in terms of 
the arrow of time and derive a lower bound by the logarithmic Sobolev inequality. These serve as 
the revealing dynamics of the information content and its limitations pertaining to nonequilibrium 
processes. 
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I. INTRODUCTION 

A fundamental aspect of physical systems being out of equilibrium lies in the existence 
of limits in information measures that the systems possess. It is known that a few measures 
of information only determined by probability distributions are closely related, but studies 
on their bounds had not been done until relatively recently when the flow vectors exist. The 
Fisher information measure, hereafter denoted as /, is an intriguing measure behind physical 
laws [l| and it appears as the basic ingredient in bounding entropy production. Specifically, 
the time derivative of the Shannon entropy dS/dt is bounded by I |2j,|3j. In this sense, there is 
a fundamental motivation in probing the interplay between physical entities and information. 
Fisher information is defined using the time-dependent probability distribution f(x,t) as 
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where (-)f denotes the average with / over the domain of phase space coordinate x, and the 
|| ■ H2 is L 2 norm. In the second expression, we use = dj dxk, (k = 1, . . . , d) with dimension 
d. At the moment of research, two different types of lower bounds are obtained through the 
upper bound on the temporal change of entropy under the Dirichlet boundary conditions. 
More specifically, the out-flux vector j(x,t) and the quantity j(x,t)lnf are required to 
vanish at the boundary in combination with the continuity equation for probability density 

^Uv.J(f, () =0. (2) 

One of these expressions was given by Nikolov and Frieden j^j as 

dt ^ 2d m dt ' [6) 

where d is the spatial dimension and (r 2 ) the mean square displacement of the particle inves- 
tigated. The fact that the entropy increase rate never reaches infinity but is appropriately 
suppressed by two characteristic quantities offers deep insight. That is, it bridges between 
the information theoretical aspect through the shape of probability distribution and the 
kinematic one through the speed of particle. The other one was expressed by Brody and 
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Meister [3J as 

§ < ^ w 

where coefficient 7 is given as {{j/f) 2 )f- Moreover, this bound represents the collaborative 
effect of I and the flow of transmitted matter. We can consider this formula as more 
versatile than Eq. ([3]), since it gives the bound if we specify a physical model (i.e., the form 
of the flux). On the other hand, a lower bound on / itself was alternatively obtained as 
(V • (j/f)) 2 /l ^ I a|- Furthermore, according to the Fickian diffusion law, j = —DVf 
with the diffusion constant D, we can easily obtain dS/dt — D ■ I ^ 0. In either case, the 
quantity / is found to play a vital role in limiting the entropy production. 



The present consideration is deeply rooted in the unidirectional characteristics of time 
(the arrow of time). Arrow of time is commonly understood as the consequence of the trans- 
formation of local information into nonlocal correlations, i.e., the production of information 
measure. In this sense, we focus on the rate of change of the Fisher information with time, 
which is termed as "Fisher information production" dl/dt in analogy to the entropy pro- 
duction dS/dt, and we will present the upper bound for it in a general setting. The present 
consideration is of high importance not only in terms of the notion of the arrow of time, 
based on /, but also from the viewpoint of the second law of thermodynamics. In Sec. II, 
we derive the lower bounds for / with and without the flow and determine the form of the 
upper bound for dl/dt. We present examples for these bounds in Sec. III. The lower bounds 
on the Fisher information production are derived in terms of Shannon entropy production 
via the logarithmic Sobolev inequality in Sec. IV. Finaly, we discuss the conclusions. 



II. NOVEL LOWER BOUNDS FOR / AND AN UPPER BOUND FOR ITS PRO- 
DUCTION 

A. A lower bound expression in presence of flux 

Since the continuity of V/ is not obvious from the onset for a given / in general, we first 
need to remark that we assume Leibniz's rule for differentiation under the integral sign; i.e., 
dl/dt = J Rl J^((V f) 2 j f)dx is assumed to be guaranteed. This implies that, in case x and t 
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are in closed intervals, (V/) 2 // is continuous for x and t. Then, ^((V/) 2 //) is also contin- 
uous. In case the integral involves an infinite domain, we require f°° J^((V f) 2 / f)dx to uni- 
formly converge with respect to t, where w irrelevant to t, such that | J^((V f) 2 / f)dx\ < e, 
Ve. 

In this context, we obtain the time derivative of the one-dimensional Fisher information, 
which yields 
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in which we have used the interchangeability f x t = ftx, since we are assuming that both f xt 
and f tx are continuous. In addition, we have substituted the continuity equation d t f = — V-j. 
Here we continue using the symbol V instead of d/dx because we want to include the 
high-dimensional case for consideration. The fact that / decreases with time and attains 
its minimum has been shown in several contexts. Indeed, the never-increasing property 
dl/dt ^ holds true for solutions of the Fokker-Planck equation [5], indicating that the 
minimum Fisher information (MFI) principle [6] induces the existence of time asymmetry 
in terms of / Q]. Accordingly, it is apparently legitimate to expect that in a wider class 
of time evolution, possessing the continuity equation, the Fisher information production is 
negative. In Sec. IV, however, we note a situation, in which this qualification is not satisfied. 
Now, imposing the decreasing /, we obtain the following inequality for the terms specified 
in Eq. (J5J 
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The last inequality follows from the Schwarz inequality. Therefore, we obtain a lower bound 
when the leftmost term of the above inequality is a positive quantity 
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This is equivalent to considering only systems with positive divergence flows. We note also 
that the inequality becomes valid when the absolute value of the leftmost part is less than 



or equal to the right hand side, i.e., | f Rl dx(V In /) 2 V • j\ ^ 21 J J Rl dx V(V • j) / f. This 
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condition limits the scope of applicability of the analysed distribution functions, and as we 
shall mention later in Sec. Ill, the Gaussian distribution lies within this applicable class. 
Consequently, the derived inequality Eq. ([7]) is yet another expression of a lower bound for 
/, which is different from the previously reported one in |4|. In the d- dimensional case, the 
Schwarz inequality gives 



y-'V(V-J; 



k=l 



v fc / 



V fc (V-J) 



v7 



where dr = dx\ ■ ■ ■ dxd- Then, we have the following inequality: 
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in which we denote the vector dl = ((dl)i, . . . ,(dX)d), whose components \\^kf / VJlh 
(k = l,...,d) constitute the Fisher information, such that |<il| 2 = /. It reflects the 
shape (geometry) property of the distribution function. On the other hand, the vector dJ 
associated with the L 2 -norm of the quantity Vfc(V • j)/y/J provides information on the 
(phase) spatial change of the flow. It is interesting to note that the bound for dS/dt by 
Brody and Meister |3j essentially stems from the application of the Schwarz inequality. 



Now, let us consider the one-dimensional case and evaluate the magnitude \dl/dt\, which 
is our second objective. Let us put the absolute values of the rightmost integrations in Eq. 
([5]) respectively as C\ and C 2 , 
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We apply the Holder inequality for the three functions fi, fi-, and ^3 
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where ||/|| p (t) = (La \f( x > t)\ p dx)^ p (1 ^ p < 00, t > 0) is the L p norm. Then, by setting 
Pi = P2 = 2 and p% = 00, the limit for C\ can be expressed as follows 
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where || • ||loo = ess. sup | • |. In conjunction with the triangle inequality for the last expression 
of Eq. ([S]), we obtain the upper bound 
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where we have put the coefficients respectively as 



a = ess. sup 
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and /3 = 2 1 
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B. Lower bound expressions in terms of the Sobolev and logarithmic Sobolev 
inequalities 

In this section, we provide the lower bounds for I depending on dimensions, when we do 
not consider the flux. Since the Fisher information captures the coarse-grained inclination 
of the distribution function, it is useful to find the operative limit of the average gradient 
in terms of the spreading of the function. 



For a function g that vanishes at infinity with its gradient in L 2 (M"), i.e., g G D 1 (IR n ), 
the Sobolev inequality with dimension n ^ 3 (e.g. |8J) reads 



(15) 



where IS™" 1 ] = 2vrf/r(n/2) is a sphere of radius 1 in R n . By substituting g = \ff we obtain 

n(n -2)2Itt 1+ ^ 
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In case of n = 2, for function g and its gradient in L 2 (R 2 ), the following inequality holds 

\\Vg\\ 2 Hl{R2) ^ C\\g\\ 2 Lq{R2) , (2^g<oo) (17) 
where the norm of the Sobolev space H l (M. 2 ) is defined as 
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and constant C depends only on q. Then, noting that ||/|| 2 = 1 by the normalization of the 
probability function, we obtain the lower bound 
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holds. By setting g = a/J, we have the inequality 

/(/) ^4(2 yT f 2 _ x). ( 2i) 
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We use this inequality in Sec. III. 

III. EXAMPLES 

A. Normal diffusion 

As the benchmark evaluation of the bounds derived here, we examine the one-dimensional 
Wiener process, where the Gaussian probability distribution function with the time- 
dependent dispersion a{t) and with the associated flow satisfying the continuity equation 
govern the system. This example has also been previously considered in the literature {3, 4]. 
The flow and the distribution function are related as j(x,t) = xfa(t)/a(t) according to 
the continuity equation and the relation j = —df/dx. By straightforward calculations, we 
obtain 

^ 6a 2 (ti r\,,.^:,.._ ^(0 




r{d x \nff Jx dx = - 2 -^ (22) 



Since the Fisher information is calculated to be / = l/a 2 (t), we can corroborate that the 
applicability condition of Eq. ([7]) is indeed satisfied, | — 2a(t) /a 3 (t)\ < 2yfl\J ((j xx / f) 2 )- 
Therefore, it is well-justified to insert these into Eq. ((?]), and thus, we obtain the lower 
bound 

1 > fcW (23) 



This provides a tighter bound compared to cr~ 2 (t) as the lower bound obtained in [4] from the 
derived inequality (i.e., / ^ (V- {j / f)) 2 f/l) therein. A possible interpretation of the origin of 
tightness is that the occurrence of flow reduces the information in a system compared to the 
case without it. In this sense, the fact that the value of the lower bound cr~ 2 (t) [4] coincides 
with that of the Fisher information /, determined only from the form of the distribution, 
does not convey limitation, at least for one-dimensional normal diffusion, and the present 
lower bound may be a preferable alternative. The coefficient 1/6 is for the one- dimensional 
case and it differs for the other dimensions. Moreover, we note that the lower bound Eq. 



(pTj) . derived from the Sobolev inequality, for one dimension is indeed found to be satisfied, 
by considering 
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and by the fact (cr - l/v 7 ^) 2 + 1/4 - 1/2tt ^ 0. 



B. Truncated Gaussian 



The use of truncated Gaussian distributions is common in statistics, econometrics, and in 
many other areas of science, where the probability density function has a cutoff from below 
or above (or both) while keeping the Gaussian form (e.g., (9j). The density function of the 
truncated normal distribution defined in x G [a, b] is 

°-^ v g 1 (25) 

where <p(x) is the standard normal distribution with mean \i and variance a 2 . $(x) denotes 
its cumulative distribution. Below, we use a time-dependent finite interval x G [—a(t),a(t)} 
by setting \i = and proceed without the scaling factor er(t)($(l) — $(—1)), derived from 
the normalization. Then, the Fisher information is calculated to be 
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(26) 



where the error function Erf(l/v^2) = 0.6826... and the positivity / > is accordingly 
maintained. The use of the truncated Gaussian in our consideration is equivalent to incor- 
poration of the flow form, same as that given in the previous example, because during the 
process, the distribution keeps the Gaussian within finite support. For the coefficients a and 
(3 defined in Eq. f|T4|) . we calculate respectively as 



a = ess. sup 
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Therefore, we find that the absolute value of the time derivative is bounded from above as 
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Since we are dealing with moving boundaries (truncation positions) here, the time derivative 
of the integral of a bivariable function, whose limits depend on time has additional terms 
(e.g., (loj) as 



dt 



— / p(x,t)dx = u(t) p(u(t) , t) — v(t)p(v(t),t) + / —p(x,t)dx 



v(t) 



v(t) 
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where p(x, t) = —f(x, t) log f(x, t) in our case and p(x, t)dx is assumed to be continuous 
within the interval of interest in both x and t. We note that the first two extra terms 
calculated as &(t)p(a(t),t) — (—&(t))p(—a(t),t) contribute only in shifting the upper bound, 
obtained in Eq. (128]) . Substituting the Gaussian form of / into p, we have p(a(t),t) = 



a 1 (t)(l/2 + log V27T + logcr(t))/v / 27re. Then, the final upper bound becomes 



dl 



dt 
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Note that by comparing Eq. ( 15D1 with Eq. (1251) . the effect of moving boundary appears as 
an increment in the bound. 



IV. A BOUND FOR dl/dt IN TERMS OF ENTROPY PRODUCTION 

In Eqs. (j3J and PJ, viewed conversely, I and its square root are respectively bounded 
from below by entropy production dS/ dt. Therefore, we can conceive the available bound in 
any form to generate change in Fisher information in terms of flow. Specifically, we develop 
an interest in obtaining a general expression of the lower bound for the Fisher information 
production dl /dt in terms of the entropy production when the systems follow the continuity 
equation d t f = —V • j with the aid of the interdependence between S and I. One may think 
that finding the lower bound on dl /dt contradicts the arrow of time employed in Sec. II. 
However, this characterization can only be true under well-organized circumstances such as 
systems in which the model follows the heat equation, although it is widely expected to hold 
as mentioned in Sec. II A. In fact, Fisher information does not necessarily decrease in time 
in certain cases. Indeed, within properly developing living cells, I is locally maximized, i.e., 



within the cell delimited by the membrane 13] . This can occur at the expense of transferring 



increased waste and disorder outside the cell. The latter ensures that the second law of 
thermodynamics is overall obeyed. Therefore, we proceed as follows. 
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Recalling that the logarithmic Sobolev inequality for the function g (e.g., js]) 

-[ \Vg(x)\ 2 dx> [ \g(x)\ 2 \n( \^.) dx + n \\g\\ 2 2 . (31) 
ft JR n JR" y \\g\\ 2 J 

Setting g = yff and by normalization of the probability density, we find 

I(f)>M"-S(f)), ( 32 ) 

where the Shannon entropy is S(f) = — J Rn f(x) In f(x)dx. Unless the function /(/) + 
&TrS(f) is a monotonically decreasing function with respect to time, there should exist 
at least a time region where its derivative becomes positive. Recalling that the identity 
dS/dt = (div(j / f)} f holds true [4] under the setting same as ours, we can rewrite it further 



as 



iS /divj\ /; .va 



it \ f i s v r- 1, 

where we have used the formula div(0A) = A ■ V0 + 0V • A for a scalar function and a 
vector A. Therefore, by differentiating Eq. ( |32l) we obtain the following lower bound 



A direct evidence for the contradiction between the property dl/dt ^ and the above- 
derived inequality can be checked in case of one-dimensional normal diffusion. That is, we 
obtain dl/dt ^ 4tt&/<j, which is finite as long as particles diffuse (i.e., a > 0). This fact 
shows the scope of the applicability. 



V. DISCUSSION 

If we specify the relation between flux vector j and distribution /, we obtain a physical 
model. In general, we can write the process as P(L)f(x,t) = j{x,t) in the one-dimensional 
case, where P(L) is a linear operator, represented by a polynomial of the differential operator 
L = d/dx with constant coefficients. Fick's law is a special case, which follows from the 
choice P(L) = —DL. Since the Fourier transform of the both sides becomes P(£)f{£) = 
j(£) when P(£) 7^ 0, (£ G M 1 ), we can have the expression of the original distribution 
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f(x,t) = (v2tt) _1 Li d£,e lx ^j(!;)/P(!;) by inverse transform. Alternatively, in terms of the 
inverse operator P _1 (L), we have the expression f(x,t) = P~ 1 (L)j(x,t). Together with the 
definition of the Fisher information in Eq. flTJ, we regard I as the result of the averaging 
function x{f) with / determined, such that 



I(t) = (*(/)>; 



(35) 



holds. If we specify the functional form of x(f) as x(f) = (j//) 2 > it is equivalent to choose 
the form of flux as Fick's law j = —df/dx. Other models are realized by fixing the form \ 
as a function of flux and distribution. Assuming that dx/dt is continuous, we have 



dl 
~dt 



dt X + f dt ]<h - 



(36) 



The upper bound of the above equation relies on the supremum, which is derived from the 
two competing terms (d t f)x an d fdtXi tmt the Schwarz' inequality at least provides the 
maxima for each term. Then, we have 

dx 



dl 




df 


dt 




dt 



dt 



(37) 



which is a desirable expression for a bound with L 2 norms because we normally require 
square integrability of distributions in physics. 



VI. CONCLUSION 



Apart from the well-known Cramer- Rao bound which asserts that I cannot exceed 
the inverse of the mean square error of a measured quantity, the fact that the bounds for 
J, \dl/dt\, and dl/dt, derived from the distribution functions, obey the laws of physics 
definitely links physics and the information contained in a physical system. The former 
(Cramer-Rao) originates from repeated active measurements (i.e., statistics), but in contrast, 
the latter originates from the flux j of a physical entity. In this context, we have presented 
new alternative upper and lower bounds for the time derivative of J. However, we neither 
have nor established a relation between the information flux and information production 



for /, whereas in nonequilibrium thermodynamics 12| based on Shannon entropy, there is 
a familiar local formulation, i.e., dS/dt = —V • J + a, where J and a denote the entropy 
flux and the entropy production, respectively. In irreversible processes, o ^ implies the 
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Boltzmann's H-theorem. A search for the counterpart may lead to a deeper understanding 
of the informational structures, inherent in physical systems. 
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